Presenting integrated building information using building models

ABSTRACT

Techniques are described for using computing devices to perform automated operations related to providing visual information of multiple types in an integrated manner about a building or other defined area. The techniques may include generating and presenting a GUI (graphical user interface) on a client device that includes a computer model of the building&#39;s interior with one or more first types of information (e.g., in a first pane of the GUI), and simultaneously presenting other types of related information about the building interior (e.g., in additional separate GUI pane(s)) that is coordinated with the first type(s) of information being currently displayed. The computer model may be a 3D (three-dimensional) or 2.5D representation generated after the house is built and showing the actual house&#39;s interior (e.g., walls, furniture, etc.), and may be displayed to a user of a client computing device in a displayed GUI with various user-selectable controls.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/681,787, filed Nov. 12, 2019 and entitled “Presenting IntegratedBuilding Information Using Three-Dimensional Building Models”, which ishereby incorporated by reference in its entirety.

TECHNICAL FIELD

The following disclosure relates generally to techniques for providingvisual information of multiple types about a defined area in anintegrated manner, such as to present a three-dimensional computer modelof an interior of an as-built building along with multiple other typesof information about the building interior in a simultaneous andcoordinated manner.

BACKGROUND

In various fields and circumstances, such as architectural analysis,property inspection, real estate acquisition and development, remodelingand improvement services, general contracting and other circumstances,it may be desirable to view information about the interior of a house,office, or other building without having to physically travel to andenter the building, including to determine actual as-built informationabout the building rather than design information from before thebuilding is constructed. However, it can be difficult or impossible toeffectively display visual information about building interiors to usersat remote locations, such as to enable a user to fully understand thelayout and other details of the interior, including under varyingconditions.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-1B are diagrams depicting an exemplary building interiorenvironment and computing system(s) for use in embodiments of thepresent disclosure, including to generate and present informationrepresenting the building interior.

FIGS. 2A-2L illustrate examples of automated operations for presentingvisual information that includes a 3D (three-dimensional) computer modelof a building's interior along with multiple additional types ofinformation about the building interior in a simultaneous andcoordinated manner.

FIG. 3 is a block diagram illustrating computing systems suitable forexecuting embodiments of one or more systems that perform at least someof the techniques described in the present disclosure.

FIG. 4 illustrates an example embodiment of a flow diagram for an ImageCapture and Analysis (ICA) system routine in accordance with anembodiment of the present disclosure.

FIGS. 5A-5B illustrate an example embodiment of a flow diagram for aFloor Map Generation Manager (FMGM) system routine in accordance with anembodiment of the present disclosure.

FIG. 6 illustrates an example embodiment of a flow diagram for aBuilding Information Integrated Presentation (BIIP) system routine inaccordance with an embodiment of the present disclosure.

DETAILED DESCRIPTION

The present disclosure describes techniques for using one or morecomputing devices to perform automated operations related to providingvisual information of multiple types about a defined area in anintegrated manner, such as information about a building and by using acomputer model of the building's interior. In at least some embodiments,the techniques include generating and presenting a GUI (graphical userinterface) on a client device that includes a visual representation of acomputer model of the building's interior with one or more first typesof information (e.g., in a first pane of the GUI), and to simultaneouslypresent other types of related information about the building interior(e.g., in one or more additional separate panes of the GUI) that iscoordinated with the first type(s) of information being currentlydisplayed. The building may, for example, be a house, the computer modelof the building's interior may be a 3D (three-dimensional) or 2.5D (twoand a half dimensional) representation that is generated after the houseis built and that shows the house's actual interior (e.g., walls,windows, doors, stairs, fireplaces, kitchen islands, cabinets, counters,lighting and/or plumbing fixtures and associated built-in elements suchas sinks and showers/baths, curtains, wall paper or paint, floorcoverings, etc.), and the types of presented information about thebuilding interior may include panorama images (e.g., 360° panoramaimages with 360° of coverage around a vertical axis), 2D(two-dimensional) perspective photos and other images, videos, aninteractive tour of inter-connected viewing/capture locations, andvarious other types of information. Additional details are includedbelow regarding the automated operations of the computing device(s)involved in the generating and presenting of the various types ofcoordinated information about a building's interior, and some or all ofthe techniques described herein may, in at least some embodiments, beperformed at least in part via automated operations of a BuildingInformation Integrated Presentation (“BHP”) system, as discussed furtherbelow.

The types of information that are presented in a simultaneous andcoordinated manner about a building's interior (and in some casessurroundings) may have various forms in various embodiments, and may beacquired in various manners. In addition, at least some of those typesof information may be associated with corresponding positions in acomputer model of the building (e.g., a 3D model with full heightinformation represented, a 2.5D model with partial representations ofheight represented, etc.). As one example, types of additionalinformation about a building may include one or more of the following,and may be associated with locations from which the information wascaptured and/or locations that are shown or otherwise represented in thecaptured information: photos or other images (e.g., 2D perspectiveimages, 360 panorama images and/or other panorama images, etc.), such asto be associated with viewing locations (also referred to at times as‘capture locations’ or ‘recording locations’ or ‘viewing/capturelocations’) within the rooms of the building where they were taken;video recordings, such as to each include a sequence of multiple imagesand to be associated with viewing locations within the rooms of thebuilding where they were taken; textual and/or audio annotations orother descriptions of particular points of interest (POIs) in thebuilding's rooms or other locations; other audio information, such asrecordings of ambient noise that are associated with recording locationswithin the rooms of the building where they were taken; a time-lapse orother accelerated video and/or animation of an area in or around a home,such as from a front door and/or front window of a house to show trafficor other information over a first period of time (e.g., 1 hour, 4 hours,8 hours, 12 hours, 24 hours, etc.) that is presented over a shorterperiod of time (e.g., 10 seconds, 30 seconds, 1 minute, 5 minutes, 10minutes, etc.); other external information from an environmentsurrounding the building, such as about nearby buildings and/orvegetation (e.g., as identified from satellite or other external imagesof the building, such as taken from the building and/or from a nearbystreet or yard; as identified from data in one or more externaldatabases or other information sources, such as street maps or othergovernment records; etc.), whether by displaying the actual images or bygenerating and displaying visual representations of particular externalelements that are identified and modeled from images or other externalinformation sources; lighting information for an interior of a buildingand/or its surroundings, such as simulated lighting for one or morerooms of a building (e.g., to simulate daylight entering the one or morerooms at one or more defined times, to show actual lighting frominterior lights in the one or more rooms at one or more defined times;etc.); in-room images for a room that are projected on the walls of theroom shown in the model; user-generated and/or ‘crowd-sourced’information provided by one or more end users about the building, suchas interior and/or exterior images, descriptions, questions, answers,etc. that are associated with particular rooms or locations withinrooms; estimated scale information such as room width, length and/orheight dimensions; geographical location and/or orientation informationfor the building; a 2D (two-dimensional) floor map of the buildinginterior, such as using an overhead schematic view (e.g., anorthographic top view); etc. Additional details are included elsewhereherein regarding types of information that may be presented about abuilding in a coordinated and simultaneous manner, such as to displayedfor a computer model of an interior of a house or other building in adisplayed GUI.

In addition, the automated operations may include controlling howmultiple types of information are presented in various embodiments in asimultaneous and coordinated manner about a building's interior (and insome cases, the building's surroundings). In at least some embodiments,such automated operations include presenting a GUI with multiple panesthat are simultaneously displayed, and with each pane includinginformation of a different type that are all related to a commonlocation or area in the building interior or other common aspect/featureof the building interior. As one non-exclusive example, a first pane maybe displaying a photo taken in a room of the building to show at leastsome of that room, a second pane may be displaying a portion of a 3Dcomputer model of the building that includes the room, and a third panemay be displaying a video taken within the room or a part of aninteractive tour of the building that includes one or more viewinglocations situated within the room—in addition, in at least someembodiments, one of the panes may be a primary pane (e.g., that islarger than other secondary panes, and/or that has user-selectablecontrols via which the user may interact with the content shown in thatpane in various manners), and the GUI may further enable the user toeasily switch content between the primary pane and one of the secondarypanes, or to otherwise change information shown in one or more of thepanes.

Various user-selectable controls may also be displayed in or otherwiseassociated with one or more of the displayed panes (e.g., with theprimary pane, and optionally in a contextual manner based on the type ofcontent displayed in the primary pane) and provide functionality to makevarious types of modifications to the displayed information, such as oneor more of the following: to change the contents of a secondary pane toa primary pane; to change the contents of the primary pane or asecondary pane to another type of information that is not currentlydisplayed, with the new information of the other type similarly beingcoordinated with the content currently displayed in the other panes; totoggle on and off one or more additional types of information that areoverlaid on the contents of a pane, such as textual and/or audiodescriptions, information about points of interest, questions and/oranswers, lighting information, etc.; to add user-specified content(e.g., a photo, a description, a question, etc.) to the content of aparticular pane and/or to a location of the building shown in that pane;etc. Additional details are included elsewhere herein regarding thepresentation of multiple types of information about a building in asimultaneous and coordinated manner, including about types ofuser-selectable controls and other user selections in a displayed GUI.

The described techniques provide various benefits in variousembodiments, including to use 3D models and/or 2.5D models and/or 2Dfloor maps of multi-room buildings and other structures (e.g., that aregenerated from images acquired in the buildings or other structures) todisplay various types of information about building interiors in acoordinated and simultaneous manner with other types of relatedinformation, including to use information about the actual as-builtbuildings (e.g., internal structural components and/or other interiorelements, nearby external buildings and/or vegetation, actual buildinggeographical location and/or orientation, actual typical weatherpatterns, etc.) rather than using information from plans on how thebuilding is designed and should theoretically be constructed. Suchdescribed techniques may further provide benefits in at least someembodiments for allowing improved automated navigation of a building bymobile devices (e.g., semi-autonomous or fully-autonomous vehicles) viause of information of various types, including to significantly reducetheir computing power used and time used to attempt to otherwise learn abuilding's layout. In addition, in some embodiments the describedtechniques may be used to provide an improved GUI in which an end usermay more accurately and quickly obtain information about a building'sinterior (e.g., for use in navigating that interior, such as via avirtual interactive tour), including in response to search requests, aspart of providing personalized information to the end user, as part ofproviding value estimates and/or other information about a building toan end user, etc. Various other benefits are also provided by thedescribed techniques, some of which are further described elsewhereherein.

For illustrative purposes, some embodiments are described below in whichspecific types of information are acquired, generated, used and/orpresented in specific ways for specific types of structures and by usingspecific types of devices—however, it will be understood that thedescribed techniques may be used in other manners in other embodiments,and that the invention is thus not limited to the exemplary detailsprovided. As one non-exclusive example, while various types ofinformation related to house interiors may be used in some situations,it will be appreciated that one or more of such information types may besimilarly used in other embodiments for other types of buildings (orother structures or layouts) separate from houses and/or for other partsof a house or other building (e.g., for external walls; surroundingyards; surrounding supplemental structures, such as a garage, shed,barn, etc.; roofs; etc.). As another example, while various types ofinformation for models of houses or other buildings may be used fordisplay to assist viewers in navigating the buildings or otherwiseunderstanding the buildings' interiors, at least some such types ofinformation may be used in other manners in other embodiments. Inaddition, the term “building” refers herein to any partially or fullyenclosed structure, typically but not necessarily encompassing one ormore rooms that visually or otherwise divide the interior space of thestructure—non-limiting examples of such buildings include houses,apartment buildings or individual apartments therein, condominiums,office buildings, commercial buildings or other wholesale and retailstructures (e.g., shopping malls, department stores, warehouses, etc.),etc. The term “acquire” or “capture” as used herein with reference to abuilding interior, viewing location, or other location (unless contextclearly indicates otherwise) may refer to any recording, storage, orlogging of media, sensor data, and/or other information related tospatial and/or visual characteristics of the building interior orsubsets thereof, such as by a recording device or by another device thatreceives information from the recording device. In addition, variousdetails are provided in the drawings and text for exemplary purposes,but are not intended to limit the scope of the invention. For example,sizes and relative positions of elements in the drawings are notnecessarily drawn to scale, with some details omitted and/or providedwith greater prominence (e.g., via size and positioning) to enhancelegibility and/or clarity. Furthermore, identical reference numbers maybe used in the drawings to identify similar elements or acts.

FIG. 1A is an example block diagram of various computing devices andsystems that may participate in the described techniques in someembodiments. In particular, an Interior Capture and Analysis (“ICA”)system (e.g., a system 160 that is executing on one or more servercomputing systems 180, and/or a system provided by application 155executing on one or more mobile image acquisition devices 185) hasacquired images 165 (e.g., 360° spherical panorama images inequirectangular format), such as with respect to one or more buildingsor other structures (not shown in FIG. 1A), and a Floor Map GenerationManager (“FMGM”) system (e.g., a system 160 that is executing on one ormore server computing systems 180, and/or a system provided byapplication 157 executing on one or more mobile image acquisitiondevices 185) has used the acquired images 165 and optionally otherinformation to generate one or more 2D floor maps 165 and/or computermodels 165 (e.g., 3D and/or 2.5D models) for the one or more buildingsor other structures. FIG. 1B shows one example of acquisition ofpanorama images for a particular house at multiple capture/viewinglocations 210, and FIGS. 2A-2L illustrate additional details about usinga computer model generated from such panorama images to control howmultiple types of information are presented in a simultaneous andcoordinated manner about the house, as discussed further below.

A BHP (Building Information Integrated Presentation) system 140 isfurther executing on one or more server computing systems to usebuilding models, maps and images 145 (e.g., acquired from theinformation 165) and/or other mapping-related information or associatedinformation (not shown) in order to control the simultaneous andcoordinated presentation of multiple types of information (includingsuch building models 145) about the house or other building. As part ofdoing so, the BIIP system may receive requests or instructions or otherinformation via computer network(s) 170 from end users of buildinginformation viewer client computing devices 175 about types ofinformation to include, before generating and providing such informationfor display on the client computing devices 175, and may furtheroptionally obtain and use supporting information supplied by BHP systemoperator users via computing devices 105 and intervening computernetwork(s) 170 to configure or modify operations of the BHP system insome embodiments. Additional details related to the automated operationof the BHP system are included elsewhere herein, including with respectto FIGS. 2A-2L and FIG. 6. In some embodiments, the ICA system(s) 160and/or FMGM system(s) 160 and/or BHP system 140 may execute on the sameserver computing system(s), such as if two or more of the systems areoperated by a single entity or are otherwise executed in coordinationwith each other (e.g., with some or all functionality of such systemsintegrated together into a larger system), while in other embodimentsthe BHP system may instead operate separately from such an ICA system(e.g., without an ICA system by instead obtaining panorama images orother images from one or more external sources) and/or may insteadoperate separately from such an FMGM system (e.g., without an FMGMsystem by instead obtaining 2D floor maps or other computer models ofbuildings from one or more external sources).

Various components of the mobile image acquisition device 185 areillustrated in FIG. 1A, including a browser 162 and/or an ICA systemapplication 155 and/or an FMGM system application 157 that are executedin memory 152 of the device 185 by one or more hardware processors 132,and including one or more imaging systems 135 to acquire visual data.The illustrated embodiment of mobile device 185 further includes one ormore sensor modules 148 that include a gyroscope 148 a, accelerometer148 b and compass 148 c in this example (e.g., as part of one or moreIMU units, not shown separately, on the mobile device), optionally a GPS(or Global Positioning System) sensor 137 or other positiondetermination sensor (not shown in this example), a display system 142,a control system 147 to control acquisition of images (e.g., to rotatethe mobile device), etc. Other computing devices/systems 105, 175 and180 may include various hardware components and stored information in amanner analogous to mobile device 185, but are not shown in this examplefor the sake of brevity, although some related further details arediscussed below with respect to FIG. 3.

In the example of FIG. 1A, the ICA system may perform automatedoperations involved in acquiring multiple images at multiple associatedviewing locations (e.g., in multiple rooms or other locations within abuilding or other structure and optionally around some or all of theexterior of the building or other structure), such as using visual dataacquired via the mobile device(s) 185, and for subsequent use ingenerating and providing a representation of an interior of the buildingor other structure. For example, in at least some such embodiments, suchtechniques may include using one or more mobile devices (e.g., a camerahaving one or more fisheye lenses or other lenses that simultaneouslycapture 360° horizontally around a vertical axis, such as to produce a360° spherical panorama image without rotation; a camera having one ormore fisheye lenses or other lenses that capture less than 360°horizontally at a given time, and that is mounted on a rotatable tripodor otherwise having an automated rotation mechanism to enable 360° imagecapture over a period of time corresponding to the rotation; a smartphone held and moved by a user, such as to rotate the user's body andheld smart phone in a 360° circle around a vertical axis; a camera heldby or mounted on a user or the user's clothing; a camera mounted on anaerial and/or ground-based drone or robotic device; etc.) to capturedata from a sequence of multiple viewing locations within multiple roomsof a house (or other building), and to optionally further capture datainvolved in movement or travel between some or all of the viewinglocations for use in linking the multiple viewing locations together,but without having distances between the viewing locations beingmeasured or having other measured depth information to objects in anenvironment around the viewing locations (e.g., without using anydepth-sensing sensors). After a viewing location's information iscaptured, the techniques may include producing a panorama image fromthat viewing location (e.g., a 360° panorama image that includes 360° ofhorizontal coverage around a vertical axis, a 360° spherical panoramaimage that further shows the surrounding room in an equirectangularformat, another type of panorama image in another format, etc.), andthen providing the panorama images for subsequent use by an FMGM systemand/or BHP system. Additional details related to embodiments of a systemproviding at least some such functionality of an ICA system are includedin U.S. Non-Provisional patent application Ser. No. 16/236,187, filedDec. 28, 2018 and entitled “Automated Control Of Image Acquisition ViaUse Of Acquisition Device Sensors”; in U.S. Non-Provisional patentapplication Ser. No. 16/190,162, filed Nov. 14, 2018 and entitled“Automated Mapping Information Generation From Inter-Connected Images”;and in U.S. Non-Provisional patent application Ser. No. 15/649,434,filed Jul. 13, 2017 and entitled “Connecting And Using Building InteriorData Acquired From Mobile Devices” (which includes disclosure of a BICAsystem that is one example embodiment of an ICA system generallydirected to obtaining and using panorama images from within one or morebuildings or other structures); each of which is incorporated herein byreference in its entirety.

In the example of FIG. 1A, the FMGM system may perform automatedoperations involved in using images acquired at multiple associatedviewing locations (e.g., in multiple rooms or other locations within abuilding or other structure and optionally around some or all of theexterior of the building or other structure) to generate a 2D floor mapfor the building or other structure and/or to generate a computer modelfor the building or other structure (e.g., a 3D model and/or a 2.5Dmodel), such as by analyzing visual information available in the images,and for providing a representation of an interior of the building orother structure (e.g., for subsequent use in controlling thesimultaneous and coordinated presentation of multiple types ofinformation about the interior of the building or other structure). Forexample, in at least some such embodiments, such techniques may includeanalyzing one or more images taken in a room to determine a shape of theroom and/or to identify inter-room passages (e.g., doorways and otheropenings in walls) into and/or out of the room. After the shapes of someor all of the rooms are determined, the techniques may further includepositioning the room shapes relative to each other to form a 2D floormap (e.g., based at least in part on connecting inter-room passagesbetween rooms, and optionally using travel or other movement informationcaptured between the viewing locations to determine relative locationsof the viewing locations with respect to each other), optionallycombined with height and/or other size information to generate a 3Dand/or 2.5D model of the building or other structure, and then providingthe generated computer model(s) and optionally 2D floor map forsubsequent use by the BHP system. Additional details related toembodiments of a system providing at least some such functionality of anFMGM system are included in U.S. Non-Provisional patent application Ser.No. 16/190,162, filed Nov. 14, 2018 and entitled “Automated MappingInformation Generation From Inter-Connected Images”; and in U.S.Provisional Patent Application No. 62/893,108, filed Aug. 28, 2019 andentitled “Automated Tools For Generating Mapping Information ForBuildings” (which includes disclosure of an MIGM system that is oneexample embodiment of a FMGM system generally directed to generatingfloor maps and other computer models for one or more buildings or otherstructures based in part of input from one or more system operatorusers); each of which is incorporated herein by reference in itsentirety.

One or more end users (not shown) of one or more building informationviewer client computing devices 175 may each further interact overcomputer networks 170 with the BHP system 140 (and optionally the ICAsystem 160 and/or FMGM system 160), such as to obtain, display andinteract with a generated computer model and/or floor map that are partof multiple types of information about the house or other building beingpresented in a simultaneous and coordinated manner in a GUI displayed ona device 175 (e.g., based at least in part on user-specifiedconditions). In addition, while not illustrated in FIG. 1A, a computermodel (or portion of it) and/or floor map may be linked to or otherwiseassociated with one or more additional types of information, such as oneor more associated and linked images or other associated and linkedinformation, including for a two-dimensional (“2D”) floor map of abuilding to be inter-linked with or otherwise associated with a separate2.5D model rendering of the building and/or a 3D floor plan modelrendering of the building, etc., and including for a computer modeland/or floor map of a multi-story or otherwise multi-level building tohave multiple associated sub-floor models or maps for different storiesor levels that are interlinked (e.g., via connecting stairway passages).Accordingly, non-exclusive examples of an end user's interactions with adisplayed or otherwise generated computer model (e.g., a 2.5D or 3Dmodel view that optionally includes images texture-mapped to walls ofthe displayed model) and/or 2D floor map of a building may include oneor more of the following: to change between a computer model view and afloor map view (collectively referred to herein as one or more mappingviews); to change between a mapping view and a view of a particularimage at a viewing location within or near the building's interior; tochange the horizontal and/or vertical viewing direction from which acorresponding subset view of (or portal into) a panorama image isdisplayed, such as to determine a portion of a panorama image in a 3Dspherical coordinate system to which a current user viewing direction isdirected, and to render a corresponding planar image that illustratesthat portion of the panorama image without the curvature or otherdistortions present in the original panorama image; etc. Additionaldetails regarding embodiments of a system providing at least some suchfunctionality of a BHP system (such as to provide or otherwise supportat least some functionality to an end user using a building informationviewer system and routine as discussed herein) are included in U.S.Non-Provisional patent application Ser. No. 15/950,881, filed Apr. 11,2018 and entitled “Presenting Image Transition Sequences Between ViewingLocations” (which includes disclosure of an ILTM system that is oneexample embodiment of a BHP system generally directed to generating anddisplaying transitions between images captured at different viewinglocations); and in U.S. Provisional Patent Application No. 62/911,959,filed Oct. 7, 2019 and entitled “Providing Simulated LightingInformation For Three-Dimensional Building Models” (which includesdisclosure of a BMLSM system that is one example embodiment of a BHPsystem generally directed to generating and displaying simulatedlighting information in a three-dimensional model of a building'sinterior); each of which is incorporated herein by reference in itsentirety. In addition, while not illustrated in FIG. 1A, in someembodiments the client computing devices 175 (or other devices, notshown) may receive and use generated computer models and/or othergenerated mapping-related information in additional manners, such as tocontrol or assist automated navigation activities by those devices(e.g., by autonomous vehicles or other devices), whether instead of orin addition to display of the generated information.

In the depicted computing environment of FIG. 1A, the network 170 may beone or more publicly accessible linked networks, possibly operated byvarious distinct parties, such as the Internet. In otherimplementations, the network 170 may have other forms, such as toinstead be a private network (such as a corporate or university network)that is wholly or partially inaccessible to non-privileged users. Instill other implementations, the network 170 may include both privateand public networks, with one or more of the private networks havingaccess to and/or from one or more of the public networks. Furthermore,the network 170 may include various types of wired and/or wirelessnetworks and connections in various situations.

FIG. 1B depicts a block diagram of an exemplary building interiorenvironment in which images are acquired and for which one or morecomputer models and/or 2D floor maps are generated, such as for furtheruse by the BHP system to control the simultaneous and coordinatedpresentation of multiple types of information about the interior of thehouse, as discussed in greater detail with respect to FIGS. 2A-2L, aswell as for use in some embodiments in otherwise presenting the computermodels and/or floor maps and/or images to users. In particular, FIG. 1Billustrates a first story of a multi-story building 198 (e.g., with apartial or full basement and/or a second story, not shown) with aninterior that was captured at least in part via multiple panoramaimages, such as by a mobile image acquisition device 185 with imageacquisition capabilities as it is moved through the building interior toa sequence of multiple viewing locations 210 (e.g., starting at viewinglocation 210A, moving to viewing location 210B along travel path 115,etc.). An embodiment of the ICA system (e.g., ICA system 160 on servercomputing system(s) 180, a copy 155 of some or all of the ICA systemexecuting on the mobile image acquisition device 185, etc.) mayautomatically perform or assist in the capturing of the datarepresenting the building interior, as well as to further analyze thecaptured data to generate panorama images to provide a visualrepresentation of the building interior, and an embodiment of the FMGMsystem (e.g., FMGM system 160 on server computing system(s) 180, a copy157 of some or all of the FMGM system executing on the mobile imageacquisition device 185, etc.) may automatically perform or assist in thegeneration of one or more computer models and/or a 2D floor maprepresenting the building interior. While such a mobile imageacquisition device may include various hardware components, such as acamera, one or more sensors (e.g., a gyroscope, an accelerometer, acompass, etc., such as part of one or more IMUs, or inertial measurementunits, of the mobile device; an altimeter; light detector; etc.), a GPSreceiver, one or more hardware processors, memory, a display, amicrophone, etc., the mobile device may not in at least some embodimentshave access to or use equipment to measure the depth of objects in thebuilding relative to a location of the mobile device, such thatrelationships between different panorama images and their viewinglocations may be determined in part or in whole based on features indifferent images, but without using any data from any such depthsensors. In addition, while geographical orientation/directionalindicator 109 is provided in FIG. 1B for reference of the viewer, themobile device and/or ICA system and/or FMGM system may not use suchabsolute directional information in at least some embodiments, such asto instead determine relative directions and distances between viewinglocations 210 without regard to actual geographical positions ordirections in such embodiments.

In operation, the mobile image acquisition device 185 arrives at a firstviewing location 210A within a first room of the building interior (inthis example, in a living room accessible via an external door 190-1),and captures a view of a portion of the building interior that isvisible from that viewing location 210A (e.g., some or all of the firstroom, and optionally small portions of one or more other adjacent ornearby rooms, such as through doors, halls, stairs or other connectingpassages from the first room). The view capture may be performed invarious manners as discussed herein, and may capture information about anumber of objects or other features (e.g., structural details) that arevisible in images captured from the viewing location—in the example ofFIG. 1B, such objects or other features throughout the house include thedoorways 190 (including 190-1 and 190-3) and 197 (e.g., with swingingand/or sliding doors), windows 196 (including 196-1, 196-2, 196-3 and196-4), corners or edges 195 (including corner 195-1 in the northwestcorner of the building 198, corner 195-2 in the northeast corner of thefirst room, corner 195-3 in the southwest corner of the first room,corner 195-4 at the northern edge of the inter-room passage between thefirst room and a hallway, etc.), furniture 191-193 (e.g., a couch 191;chair 192; table 193; etc.), pictures or paintings or televisions orother hanging objects 194 (such as 194-1 and 194-2) hung on walls, lightfixtures, various built-in appliances or fixtures (not shown), etc. Theuser may also optionally provide a textual or auditory identifier to beassociated with a viewing location, such as “living room” for the roomincluding viewing locations 210A and/or 210B and/or identifiers “entry”and “living room -NE” for the viewing locations 210A and 210B,respectively, while in other embodiments the ICA system and/or FMGM mayautomatically generate some or all such identifiers (e.g., byautomatically analyzing video and/or other recorded information for abuilding to perform a corresponding automated determination, such as byusing machine learning) or the BHP system may determine such identifiers(e.g., based at least in part on input from BHP system operator usersand/or end users) or the identifiers may not be used.

After the first viewing location 210A has been adequately captured, themobile device 185 may move or be moved to a next viewing location (suchas viewing location 210B), optionally recording movement data such asvideo and/or other data from the hardware components (e.g., from one ormore IMUs, from the camera, etc.) during movement between the viewinglocations. At the next viewing location, the mobile device may similarlycapture a panorama image from that viewing location. This process mayrepeat for some or all rooms of the building and optionally external tothe building, as illustrated for viewing locations 210C-210J in thisexample. The acquired panorama images for each viewing location may befurther analyzed, including in some embodiments to render or otherwiseplace each panorama image in an equirectangular format, whether at thetime of image capture or later.

Various details are provided with respect to FIGS. 1A-1B, but it will beappreciated that the provided details are non-exclusive examplesincluded for illustrative purposes, and other embodiments may beperformed in other manners without some or all such details.

FIGS. 2A-2L illustrate examples of presenting a GUI that simultaneouslydisplays multiple types of information about an interior of a buildingin an integrated and coordinated manner, such as for the building 198and using images and their viewing/capture locations 210 discussed inFIG. 1B.

In particular, FIG. 2A illustrates an example embodiment of a GUI withvarious information 204 a, which is displayed in this example on amobile client device 205 (e.g., a smart phone or tablet), and whichshows multiple types of information about a common area of examplebuilding 198. The GUI in this example includes a primary pane 207 andtwo secondary panes 208 and 209, with each of the panes showinginformation of a different type about the same area of the example house198 shown in FIG. 1B. The primary pane 207 is showing an image takenfrom viewing location 210A in the living room in this example, with thedirection of the image being in the northeast direction (as shown byvisual indicator 109 a overlaid on the primary pane, as well as thevisual indicator 219 a shown in secondary pane 209). In this example,the primary pane 207 also includes a header 206 that provides adescription of the type of content shown (here, an image, as illustratedby the “photo gallery” label), as well as optionally having anadditional label specific to the photo or other image being displayed(here, “Living Room #1”).

The primary pane 207 may also have one or more user-selectable controlsthat enable the end user (not shown) to modify information beingdisplayed in the GUI, such as control 218 a in the header area 206 tochange the type of information being shown in the primary pane to adifferent type of content (e.g., by cycling through each of theavailable types of content). In addition, other possible user-selectablecontrols 201 are overlaid on the primary pane, such as a toggle control203 c to show or hide information about points of interest that arepresent in the current image, a toggle control 203 d to show or hide adescription of the area of the house shown by the image, a togglecontrol 203 e to show or hide information about questions and answerscorresponding to the area of the house shown in the image and tooptionally allow the end user to supply one or more additional questionsor answers, a toggle control 203 b to switch to a so-called “AR” (oraugmented reality) mode in which tilting and rotating the body of themobile device causes the displayed image to move accordingly (e.g., ifthe image shown in the primary pane is a 360° panorama image with only asubset of it currently shown, such that tilting to the left causes anadditional portion of the panorama image to the left of the visible areato be shown, and with the control 203 b not being shown if the contentis not a panorama image), a selection control 203 a to switch to anotherimage in the photo gallery (e.g., to cycle through all availableimages), etc.—as discussed in greater detail elsewhere herein, in atleast some embodiments and situations, some or all of the displayeduser-selectable controls may be contextual based on a type of contentcurrently shown in the primary pane (in this example, associated withimage content).

In addition to the image shown in the primary pane of the northeastcorner of the living room of the example house, the secondary panes 208and 209 show other types of content about the same location or area ofthe example house. In particular, the secondary pane 209 in this exampleshows a portion of a 3D computer model of the house, and in particularshows a portion of the computer model that includes the northeast cornerof the living room, as well as optionally including some or all of otherparts of the same floor—as previously noted, the computer model isoverlaid in this example with the visual indicator 219 a to illustratewhere the image shown in the primary pane was captured, and theorientation of the camera that captured the image. In addition, thesecondary pane 208 in this example shows a portion of an interface to aninteractive virtual tour of the house, with the interactive tour havinga plurality of inter-connected viewing/capture locations at which imagesand/or other information that are available for viewing or otherpresentation were captured. In this example, the interface includes animage that was also captured from viewing location 210A (e.g., apanorama image captured from that viewing location), with the interfacefurther including visual indicators 214 b and 214 c that each correspondto other viewing locations different from viewing location 210A (inparticular, to viewing locations 210B and 210C of FIG. 1B,respectively)—as discussed in greater detail with respect to FIG. 2C,such visual indicators 214 may be user-selectable controls in at leastsome situations (e.g., when the interface to the interactive tour isshown in the primary pane) such that selection of one of the controlschanges the image displayed to change to one from the viewing locationassociated with the selected control.

It will be appreciated that a variety of types of user interactions mayoccur with example embodiments of the GUI, such as to enable an end user(not shown) to select a secondary pane and cause its content to be movedto the primary pane, with new contents of that secondary pane beingselected to correspond to the new primary pane contents (e.g., to movethe contents that were in the primary pane to that secondary pane, toselect a new content type to display in the secondary pane, etc.). Otheroptional user-selectable controls 202 are further shown in this examplefor reference purposes, although they may not be visible in someembodiments when three GUI panes are displayed as shown, such as if thecontrols 202 are associated with a different part of the GUI asdiscussed in greater detail with respect to FIG. 2F.

FIG. 2B continues the example of FIG. 2A, and in particular correspondsto the end user having selected the control 203 a in FIG. 2A to changethe image that is displayed in the primary pane to another image fromthe photo gallery. In this example, the new image displayed in theprimary pane 207 as part of the updated information 204 b in the updatedGUI is also taken from the same viewing location 210A, but in anorthwesterly direction (as illustrated by the updated visual indicator219 b shown in the 3D computer model of the secondary pane 209). In thisexample, the contents of the secondary pane 209 continue to be the sameportion of the 3D computer model as in FIG. 2A (with the updated visualindicator 219 b), which continues to correspond to the area of the houseshown in the primary pane of FIG. 2B, and the contents of the secondarypane 208 continue to be the same portion of the interactive tourinterface that correspond to the content of FIG. 2B as well (e.g., sincethe image in FIG. 2B continues to be taken from the same viewing/capturelocation as in FIG. 2A). It will be appreciated that the header portion206 of the primary pane 207 is also updated in FIG. 2B to reflect thenew image being shown in the primary pane, although in some embodimentssuch image-specific labels may not be used. It is also noted that theuser-selectable controls 202 mentioned in FIG. 2A are not shown in FIG.2B.

In addition, FIG. 2B further illustrates additional information that hasbeen overlaid on the primary pane 207, based on selection by the enduser of the user-selectable controls 203 c, 203 d and 203 e in thisexample, although it will be appreciated that the end user may insteadselect only zero, one, or two of those three controls in othersituations. In this example, selection of the user-selectable control203 c has caused several visual indicators 212 for points of interest inthe room to be illustrated, with visual indicator 212 b being currentlyselected and having a corresponding textual comment in the area 213 ofthe primary pane (to comment about paint on the walls of the room).Other visual POI indicators in this example include 212 a on thewest-facing picture window, 212 d on the north-facing window, and 212 con the overhead track lighting on the ceiling. While the selected visualindicator 212 b has associated text that is displayed in this example,other visual indicators for POIs may have other types of informationassociated, such as if the window visual indicators 212 a and/or 212 dhave images and/or video associated with them, such as to show images orvideos looking out the window (e.g., a time-lapse video over a 24-hourperiod of the exterior of the house from a window or door, such as toshow road traffic or people traffic outside over that time period). Inaddition to the activation of the control 203 c, the user-selectablecontrol 203 d has also been activated to provide an audio description ofthe area shown in the image, with the visual information 211 shown inthis example representing audio information that may be audiblypresented in response to the selection of that control (e.g., instead ofhaving a textual representation of the information as shown in thisexample, or instead in addition to the textual information, such as if aclosed captioning option is selected). Furthermore, the user-selectablecontrol 203 b corresponding to questions and answers is also selected,causing additional information in the area 213 to be shown, such as aquestion from another end user and a corresponding answer from a realestate listing agent for the house, as well as further user-selectablecontrols to allow the end user to ask a question to be answered, submita comment for display to others, or to cycle through other existingquestions and comments. It will be appreciated that illustrated types ofinformation may be presented in other manners in other embodiments, ormay not be shown.

FIG. 2C continues the examples of FIGS. 2A-2B, and illustrates anexample of the end user interacting with the GUI displayed in FIG. 2A toinitiate a switch of the content between the primary pane 207 and thesecondary pane 208, such that the primary pane 207 in the updatedinformation 204 c of the updated GUI of FIG. 2C is the interface to theinteractive tour of the house, with the visual indicators 214 b and 214c in FIG. 2C now being selectable by the user to change the currentviewing location, as further illustrated with respect to FIG. 2D. In atleast some embodiments and situations, the information displayed in thesecondary pane 209 would not change in FIG. 2C (other than updating thevisual indicator 219 c), such as due to the content of the primary panecontinuing to correspond to the same portion of the 3D computer modelthat was shown in FIGS. 2A and 2B—in this example, however, theinformation in the secondary pane 209 has been updated for the purposesof illustration to show the entire 3D computer model for the currentfloor of the house. In addition, contents of the secondary pane 208 havebeen updated in this example to show the image that was previouslypresent in the primary pane in FIG. 2A. It will be appreciated that theheader portion 206 of the primary pane 207 is also updated in FIG. 2C toreflect the new type of content being shown in the primary pane.

FIG. 2D continues the examples of FIGS. 2A-2C, and in this examplecorresponds to the end user having selected the user-selectable visualindicator 214 b in FIG. 2C to change the current viewing location forthe interactive tour. In response to that selection, the backgroundimage used for the interactive tour interface in the primary pane 207 ofFIG. 2D has changed to the updated information 204 d shown in theprimary pane for the updated GUI, corresponding to an image taken fromthe viewing location 210B in the northeast part of the living room, andthe user-selectable visual indicators in the primary pane have similarlybeen updated, with the direction of the previous indicator 214 cchanging to point to the direction of the hallway from the newbackground image that is shown, and with the previous visual indicator214 b for the northeast living room viewing location 210B (which is nowthe current viewing location) being changed to the new user-selectablevisual indicator 214 a for the entry viewing location 210A in the livingroom (which was the previous viewing location). In addition, a newvisual indicator 203 f has been added to reflect an alternative mannerfor the end user to adjust the panorama image shown as the background ofthe primary pane, such as to manually drag the image up, down, leftand/or right to display other parts of the panorama image that are notcurrently visible in the subset shown in the primary pane, as discussedfurther in FIG. 2E. As in the previous examples, the visual indicator219 d in the 3D computer model of the secondary pane 209 has beenupdated to correspond to the location and orientation at which the newbackground image shown in the primary pane was captured. In addition,while the content previously shown in the secondary pane 208 of FIG. 2Cmay in some embodiments and situations continue to be shown in thesecondary pane 208 in FIG. 2D, in this example the content has beenchanged for illustrative purposes to provide content of another typethat corresponds to the same area of the house, which in this example isa video of at least that portion of the house. In this example, thevideo in the secondary pane 208 includes a visual indicator 203 g of aplay button for the video, although the visual indicator may not beuser-selectable in at least some embodiments until the video content ismoved to the primary pane.

FIG. 2E continues the examples of FIG. 2A-2D, and in this exampleillustrates the effects of the end user having used the control 203 f ofFIG. 2D to change the subset of the background panorama image (e.g., a360° panorama image) that is shown by rotating approximately 180°, so asto now point toward the southwest corner of the living room from theviewing location 210B in the northeast corner of the living room, asshown in the updated information 204 e of the updated GUI. In thisexample, the user-selectable visual indicators 214 c and 214 a (forviewing locations 210C and 210A, respectively) have been updated tocorrespond to the positions of those viewing locations relative to thecontent shown in the background image, and the visual indicator 219 e inthe 3D computer model of the secondary pane 209 has been similarlyupdated. While the contents of the secondary pane 208 may be unchangedin this situation, the image in the photo gallery of secondary pane 208has been changed in this example to an image that is closest to that ofthe current background image in the primary pane (so as to maintain acoordinated display of information that is integrated with the primarypane), which in this example is of the window of the west wall of theliving room. In addition, this example further illustrates thatadditional types of information may be overlaid on images in at leastsome situations, such as in this example to overlay simulated lightinginformation 216 a on that image (although a correspondinguser-selectable control to initiate that display is not shown in thisexample). It will be appreciated that additional overlay information,such as that of the simulated lighting, may be controlled by the enduser in various manners, including as is discussed in greater detailwith respect to FIG. 2F.

FIG. 2F continues the examples of FIGS. 2A-2E, and illustrates updatedinformation 204 f in the updated GUI in which the secondary panes 208and 209 are not shown, but the additional user-selectable controls 202are included (e.g., based on the primary pane 207 being changed to showthe 3D computer model of the house). In this example, the headerinformation 206 is updated to include an additional user-selectablecontrol 218 b that allows the end user to select which floor of thehouse is shown, with the house in this example having an additionalbasement floor accessible via the stairs (as illustrated further withrespect to FIG. 2G). In this example, the 3D computer model includesillustrations of the positions of the viewing/capture locations for theinteractive tour, with the visual indicator 216 a being added tocorrespond to the current viewing location that was last selected. Thisexample further includes a user-selectable control 216 b that allows anend user to orient the computer model of the house in different mannersaccording to different specified criteria, such as to have a top of theprimary pane correspond to magnetic or geographic north, have the top orbottom of the primary pane correspond to the main entry of the house, toprovide a best fit of the 3D computer model to the current size andshape of the primary pane, etc. In addition, the user-selectablecontrols 202 may enable various types of additional information to beoverlaid on the floor map or to otherwise invoke additionalfunctionality to be provided, such as one or more of the following:control 202 a activates a default mode that allows the end user to moveto a detailed view of a particular room or viewing location by selectingthat room or viewing location on the 3D computer model; control 202 bshows additional descriptive information of one or more types on thecomputer model; control 202 c shows measurements or other scaleinformation on the 3D computer model; control 202 d shows simulatedsunlight on the 3D computer model for specified conditions (e.g., one ormore times of day and/or times of year); control 202 e shows actualinterior lighting, optionally under specified conditions; control 202 fshows information about a surrounding environment of the building (withFIG. 2G providing one example); control 202 g provides sound recordingsfrom one or more locations of the building (e.g., of ambient sound atspecified times); control 202 z allows virtual objects to be added tothe 3D computer model and/or surfaces in the 3D computer model to havevirtual changes (e.g., to change color, texture etc. of walls, floors,object surfaces, etc.).

FIG. 2G continues the examples of FIGS. 2A-2F, and illustrates anexample of the GUI being updated with information 204 g to again includeonly a single primary pane 207, which in this example has no header orseparate area for user-selectable controls—in addition, the informationbeing displayed has been adjusted to reflect that the device 205 hasbeen rotated to a landscape orientation. In this example, the end userhas selected to display information about the surroundings of thebuilding, which in this example includes a street to the west of thehouse, with two additional houses across the street being shown, as wellas additional information on the house's property that includes twotrees, sidewalks, a driveway and garage, and bushes or other shrubs onthe south side of the house (as indicated by the visual indicator 109g). This example further illustrates a deck to the east of the house,and includes visual information about a basement existing below groundlevel, although details of the interior of the basement are not shown inthis example. The vegetation and surrounding buildings are shown in thisexample using simplified geometrical shapes, such as to enable the useof such shapes to simulate shading (not shown) if a simulated lightinginformation option is selected for the display, although actual imagesof surroundings may instead be used in other embodiments and situations.It will be appreciated that a variety of other types of information maybe shown about the surroundings of the building, including images and/orvideo from ground level and/or an aerial view, as well as additionalpoints of interest in descriptive information about the surroundingneighborhood.

FIG. 2H continues the examples of FIGS. 2A-2G, and illustrates anexample of the GUI similar to that of FIG. 2F, but updated withinformation 204 h corresponding to selection of control 202 h by the enduser. In this example, the control 202 h corresponds to updating thedisplayed floor plan of the house to visually show information aboutspaces of the house that are public (e.g., living room, kitchen, diningroom, hallways, stairs, etc.) and that are private (e.g., bathrooms,bedrooms, etc.), such as displaying one or both types of spaces withspecified colors and/or patterns. In this example, a legend 221 h isdisplayed to show options 222 h for patterns to use, with the displayedfloor plan having corresponding displayed information (e.g., via apublic/private visual layer that is overlaid on the floor plan). Thepublic and private spaces of the house may be determined in variousmanners in various embodiments, such as automatically (e.g., based onpreviously determined room types, based on analysis of images) and/orbased on user annotations or other input (e.g., from one or more systemoperator users of the BHP and/or ICA systems, from end users, etc.).

FIG. 2I continues the examples of FIGS. 2A-2H, and illustrates anexample of the GUI similar to that of FIG. 2H, but instead being updatedwith information 204 i (instead of information 204 h) corresponding toselection of control 202 i by the end user. In this example, the control202 i corresponds to updating the displayed floor plan of the house tovisually show icon information about types of rooms of the house (e.g.,instead of the room type textual labels shown in FIG. 2H). In thisexample, a legend 221 i is displayed to show icons 222 i to use fordifferent types of rooms, with the displayed floor plan havingcorresponding displayed icons (e.g., via a room type icon visual layerthat is overlaid on the floor plan, such as instead of or in addition toa room type textual label visual layer). As previously discussed, typesof rooms may be determined in various manners in various embodiments,such as automatically (e.g., based on analysis of images and/or use ofmachine learning techniques) and/or based on user annotations or otherinput (e.g., from one or more system operator users of the BHP and/orICA systems, from end users, etc.).

FIG. 2J continues the examples of FIGS. 2A-2I, and illustrates anexample of the GUI similar to that of FIG. 2I, but being updated withinformation 204 j corresponding to selection of control 202 j by the enduser. In this example, the control 202 j corresponds to updating thedisplayed floor plan of the house to visually show examples ofinhabitant flow pattern information 221 j corresponding to typical orcommon movement patterns of people (e.g., in addition to room typeicons, while in other embodiments may be shown without room type labelor icon information). In this example, the flow pattern information maybe displayed, for example, via a flow pattern visual layer that isoverlaid on the floor plan, such as instead of or in addition to one ormore other visual layers. Such flow pattern information may bedetermined in various manners in various embodiments, such asautomatically (e.g., based on analysis of video taken over time withinthe house interior, based on other tracking of user movements within thehouse interior, based on analysis of images to determine open lanes orspaces for user movement, etc.) and/or based on user annotations orother input (e.g., from one or more system operator users of the BHPand/or ICA systems, from end users, etc.).

FIG. 2K continues the examples of FIGS. 2A-2J, and illustrates anexample of the GUI similar to that of FIG. 2H, but updated withinformation 204 k corresponding to selection of control 202 k by the enduser. In this example, the control 202 k corresponds to updating thedisplayed floor plan of the house to visually show information aboutspaces of the house that are likely to be of most interest to the enduser, such as displaying spaces corresponding to different levels oflikely user interest with different specified colors and/or patterns. Inthis example, a legend 221 k is displayed to show patterns 222 k forpatterns to use, with the displayed floor plan having correspondingdisplayed information (e.g., via a user-specific personalized visuallayer that is overlaid on the floor plan). The areas of the house thatare likely to be of interest to the end user may be determined invarious manners in various embodiments, such as automatically (e.g.,based on tracking a quantity of time and/or a length of time that theend user has previously spent viewing different parts of this house,based on an analysis of corresponding types of information for the enduser from multiple other houses, etc.) and/or based on user annotationsor other input (e.g., from previously specified preferences of the enduser; from previous indications of the end user of particular parts ofthe house that are of interest, such as by flagging or pinningparticular rooms and/or images; etc.).

FIG. 2L continues the examples of FIGS. 2A-2K, and illustrates anexample of the GUI similar to that of FIG. 2F, but updated withinformation 204 l corresponding to selection of control 202 l by the enduser. In this example, the control 202 l corresponds to providinginteractive functionality for the end user to specify a length on thedisplayed floor plan of the house (e.g., by dragging a line 223 l, byselecting two end points, etc.) and receive information about thedistance of that length (e.g., displayed distance information 224 l).The distance may be automatically determined in various manners invarious embodiments, such as based on previous and/or concurrentanalysis of images of the house interior to determine sizes of known orunknown objects, as discussed in greater detail elsewhere herein. Inaddition, while the distance functionality is provided using a 3D floorplan in this example, similar functionality may be provided on othermedia or types of information in other embodiments, such as on displayedpanorama or perspective images, during interactive virtual tours ofinter-linked viewing locations, etc.

Various details have been provided with respect to FIGS. 2A-2L, but itwill be appreciated that the provided details are non-exclusive examplesincluded for illustrative purposes, and other embodiments may beperformed in other manners without some or all such details. Forexample, while not illustrated in the examples of FIGS. 2A-2L, the GUIcould have other forms in other embodiments, such as to have otheruser-selectable and/or user-modifiable controls (whether instead of orin addition to the illustrated controls), and/or to have particularcontrols be accessed and used in other manners, and/or to be displayedor otherwise presented on other types of devices (e.g., desktop orlaptop computers, tablet computers, etc.), and/or to be displayed orotherwise presented in other types of formats and layouts (e.g., to havemore or less GUI panes, to have different layouts and/or sizes of GUIpanes, to display multiple types of information simultaneously withoutusing GUI panes like those of the examples, etc.). In addition,additional types of functionality related to displaying or otherwisepresenting building interior information may be provided in otherembodiments. Various other changes to the GUI may be further made inother embodiments.

FIG. 3 is a block diagram illustrating an embodiment of one or moreserver computing systems 300 executing an implementation of a BHP system340, optionally one or more server computing systems 380 executing animplementation of an ICA system 389, and optionally one or more servercomputing systems 370 executing an implementation of an FMGM system379—the server computing system(s) and BHP and/or FMGM and/or ICAsystems may be implemented using a plurality of hardware components thatform electronic circuits suitable for and configured to, when incombined operation, perform at least some of the techniques describedherein. In the illustrated embodiment, each server computing system 300includes one or more hardware central processing units (“CPUs”) or otherhardware processors 305, various input/output (“I/O”) components 310,storage 320, and memory 330, with the illustrated I/O componentsincluding a display 311, a network connection 312, a computer-readablemedia drive 313, and other I/O devices 315 (e.g., keyboards, mice orother pointing devices, microphones, speakers, GPS receivers, etc.).Each server computing system 380 and 370 may have similar components,although only one or more hardware processors 381 and 371, memory 387and 377, storage 385 and 375, and I/O components 382 and 372,respectively, are illustrated for the sake of brevity.

The server computing system(s) 300 and executing BHP system 340, andserver computing system(s) 380 and executing ICA system 389 if present,and server computing system(s) 370 and executing FMGM system 379 ifpresent, may communicate with each other and with other computingsystems and devices in this illustrated embodiment via one or morenetworks 399 (e.g., the Internet, one or more cellular telephonenetworks, etc.), such as to interact with user client computing devices390 (e.g., used to view a GUI that simultaneously displays multipletypes of information about an interior of a building in an integratedand coordinated manner, or to otherwise present information about abuilding), and/or one or more mobile image acquisition devices 360(e.g., used to acquire images and optionally other information forbuildings or other environments to be modeled), and/or optionally othernavigable devices 395 that receive and use computer models and/or otherbuilding information (e.g., 2D floor maps) for navigation purposes(e.g., for use by semi-autonomous or fully autonomous vehicles or otherdevices). In other embodiments, some of the described functionality maybe combined in less computing systems, such as to combine the ICA system389 and the image acquisition functionality of device(s) 360 in a singlesystem or device (e.g. via the optional ICA application 368 executing inmemory 367 of the mobile device 360), to combine the BHP system 340and/or the ICA system 389 and/or the FMGM system 379 in a single systemor device, to combine the BHP system 340 and the ICA system 389 and theFMGM system 379 and the image acquisition functionality of device(s) 360in a single system or device, etc.

In the illustrated example of FIG. 3, an embodiment of the BHP system340 executes in memory 330 of the server computing system(s) 300 inorder to perform at least some of the described techniques, such as byusing the processor(s) 305 to execute software instructions of thesystem 340 in a manner that configures the processor(s) 305 andcomputing system 300 to perform automated operations that implementthose described techniques. The illustrated embodiment of the BHP systemmay include one or more components, not shown, to each perform portionsof the functionality of the BHP system, and the memory may furtheroptionally execute one or more other programs 335—as one specificexample, a copy of the ICA and/or FMGM systems may each execute as oneof the other programs 335 in at least some embodiments, such as insteadof or in addition to the ICA system 389 on the server computingsystem(s) 380 and the FMGM system 379 on the server computing system(s)370. The BHP system 340 may further, during its operation, store and/orretrieve various types of data on storage 320 (e.g., in one or moredatabases or other data structures), such as various types of userinformation 322, images and other media information 325 acquired from orabout a building (e.g., received from ICA system 389, to provide tousers of client computing devices 390 for display; etc.), generatedcomputer models and optionally floor maps and other associated mappinginformation 326 (e.g., received from FMGM system 379, such as generatedand saved 2.5D and/or 3D models, 2D floor maps, etc.), other types ofbuilding information 324 (e.g., annotations and other descriptions,information about points of interest, information about surroundingbuildings and/or vegetation and/or other exterior information for thebuilding, etc.; as well as information from end users, such asquestions; information from BHP system operator users or otherauthorized users, such as answers to questions; etc.), and/or varioustypes of optional additional information 328 (e.g., various analyticalinformation related to analysis of other building information and/or enduser activities).

In addition, an embodiment of the ICA system 389 executes in memory 387of the server computing system(s) 380 in the illustrated embodiment inorder to perform automated operations related to acquiring images ofbuilding interiors (and optionally exteriors of buildings, includingtheir surroundings), such as by using the processor(s) 381 to executesoftware instructions of the system 389 in a manner that configures theprocessor(s) 381 and computing system 380 to perform such automatedoperations. Similarly, an embodiment of the FMGM system 379 executes inmemory 377 of the server computing system(s) 370 in the illustratedembodiment in order to perform automated operations related togenerating computer models and optionally floor maps of buildinginteriors, such as by using the processor(s) 371 to execute softwareinstructions of the system 379 in a manner that configures theprocessor(s) 371 and computing system 370 to perform such automatedoperations. The illustrated embodiments of the ICA and/or FMGM systemsmay each include one or more components, not shown, to each performportions of the functionality of their respective ICA or FMGM system,and the respective computer memories may further optionally execute oneor more other programs (not shown). The ICA system 389 and/or FMGMsystem 379 may further, during their operation, store and/or retrievevarious types of data on storage 385 or 375, respectively (e.g., in oneor more databases or other data structures), such as acquired images386, generated computer models 376 (e.g., generated and saved 2.5Dand/or 3D models) and optionally floor maps and other associatedinformation 376 (e.g., building and room dimensions for use withassociated floor plans, additional images and/or annotation information,various analytical information related to presentation or other use ofone or more building interiors or other environments, etc.)—while notillustrated in FIG. 3, the ICA and/or FMGM systems may further store anduse additional types of information, such as about system operator usersof the respective systems, metadata about acquisition of images to beanalyzed, etc.

Some or all of the user client computing devices 390 (e.g., mobiledevices), mobile image acquisition devices 360, optional other navigabledevices 395 and other computing systems (not shown) may similarlyinclude some or all of the same types of components illustrated forserver computing system 300. As one non-limiting example, the mobileimage acquisition device(s) 360 are each shown to include one or morehardware CPU(s) 361, I/O components 362, storage 365, and memory 367,with one or both of a browser and one or more client applications 368(e.g., an application specific to the FMGM system and/or ICA systemand/or BHP system) executing within memory 367, such as to participatein communication with the BHP system 340, ICA system 389, FMGM system379 and/or other computing systems—the devices 360 each further includeone or more imaging systems 364 and IMU hardware sensors 369, such asfor use in acquisition of images and associated movement/travel data ofthe device 360. While particular components are not illustrated for theother navigable devices 395 or other computing systems 390, it will beappreciated that they may include similar and/or additional components.

It will also be appreciated that computing systems 300, 370 and 380 andthe other systems and devices included within FIG. 3 are merelyillustrative and are not intended to limit the scope of the presentinvention. The systems and/or devices may instead each include multipleinteracting computing systems or devices, and may be connected to otherdevices that are not specifically illustrated, including via Bluetoothcommunication or other direct communication, through one or morenetworks such as the Internet, via the Web, or via one or more privatenetworks (e.g., mobile communication networks, etc.). More generally, adevice or other computing system may comprise any combination ofhardware that may interact and perform the described types offunctionality, optionally when programmed or otherwise configured withparticular software instructions and/or data structures, includingwithout limitation desktop or other computers (e.g., tablets, slates,etc.), database servers, network storage devices and other networkdevices, smart phones and other cell phones, consumer electronics,wearable devices, digital music player devices, handheld gaming devices,PDAs, wireless phones, Internet appliances, and various other consumerproducts that include appropriate communication capabilities. Inaddition, the functionality provided by the illustrated systems 340and/or 379 and/or 389 may in some embodiments each be distributed invarious components, some of the described functionality of the systems340 and/or 379 and/or 389 may not be provided, and/or other additionalfunctionality may be provided.

It will also be appreciated that, while various items are illustrated asbeing stored in memory or on storage while being used, these items orportions of them may be transferred between memory and other storagedevices for purposes of memory management and data integrity.Alternatively, in other embodiments some or all of the softwarecomponents and/or systems may execute in memory on another device andcommunicate with the illustrated computing systems via inter-computercommunication. Thus, in some embodiments, some or all of the describedtechniques may be performed by hardware means that include one or moreprocessors and/or memory and/or storage when configured by one or moresoftware programs (e.g., by the BHP system software 340 executing onserver computing systems 300 and/or on devices 360, by the ICA systemsoftware 389 executing on server computing systems 380, by the FMGMsystem software 379 executing on server computing systems 370, etc.)and/or data structures, such as by execution of software instructions ofthe one or more software programs and/or by storage of such softwareinstructions and/or data structures, and such as to perform algorithmsas described in the flow charts and other disclosure herein.Furthermore, in some embodiments, some or all of the systems and/orcomponents may be implemented or provided in other manners, such as byconsisting of one or more means that are implemented partially or fullyin firmware and/or hardware (e.g., rather than as a means implemented inwhole or in part by software instructions that configure a particularCPU or other processor), including, but not limited to, one or moreapplication-specific integrated circuits (ASICs), standard integratedcircuits, controllers (e.g., by executing appropriate instructions, andincluding microcontrollers and/or embedded controllers),field-programmable gate arrays (FPGAs), complex programmable logicdevices (CPLDs), etc. Some or all of the components, systems and datastructures may also be stored (e.g., as software instructions orstructured data) on a non-transitory computer-readable storage mediums,such as a hard disk or flash drive or other non-volatile storage device,volatile or non-volatile memory (e.g., RAM or flash RAM), a networkstorage device, or a portable media article (e.g., a DVD disk, a CDdisk, an optical disk, a flash memory device, etc.) to be read by anappropriate drive or via an appropriate connection. The systems,components and data structures may also in some embodiments betransmitted via generated data signals (e.g., as part of a carrier waveor other analog or digital propagated signal) on a variety ofcomputer-readable transmission mediums, including wireless-based andwired/cable-based mediums, and may take a variety of forms (e.g., aspart of a single or multiplexed analog signal, or as multiple discretedigital packets or frames). Such computer program products may also takeother forms in other embodiments. Accordingly, embodiments of thepresent disclosure may be practiced with other computer systemconfigurations.

FIG. 4 illustrates an example flow diagram of an embodiment of an ICASystem routine 400. The routine may be performed by, for example, theICA system 160 of FIG. 1A, the ICA system 389 of FIG. 3, and/or the ICAsystem described with respect to FIGS. 1B-2L and as otherwise describedherein, such as to acquire images (e.g., 360° spherical panorama images)at viewing locations within buildings or other structures, such as foruse in subsequent generation of related floor maps and/or other mappinginformation. While portions of the example routine 400 are discussedwith respect to acquiring particular types of images at particularviewing locations, it will be appreciated that this or a similar routinemay be used to acquire video or other data (e.g., audio), whetherinstead of or in addition to such images. In addition, while theillustrated embodiment acquires and uses information from the interiorof a target building, it will be appreciated that other embodiments mayperform similar techniques for other types of data, including fornon-building structures and/or for information external to one or moretarget buildings of interest. Furthermore, some or all of the routinemay be executed on a mobile device used by a user to acquire imageinformation, and/or by a system remote from such a mobile device.

The illustrated embodiment of the routine begins at block 405, whereinstructions or information are received. At block 410, the routinedetermines whether the received instructions or information indicate toacquire data representing a building interior, and if not continues toblock 490. Otherwise, the routine proceeds to block 412 to receive anindication from a user of a mobile image acquisition device to begin theimage acquisition process at a first viewing location. After block 412,the routine proceeds to block 415 in order to perform viewing locationimage acquisition activities in order to acquire an image (e.g., a 360°panorama image) for the viewing location in the interior of the targetbuilding of interest, optionally via one or more fisheye lenses on themobile device to provide horizontal coverage of at least 360° around avertical axis. As one non-exclusive example, the mobile imageacquisition device may be a rotating (scanning) panorama camera equippedwith a fisheye lens, such as a 180° fisheye giving a full sphere at 360°rotation. The routine may also optionally obtain annotation and/or otherinformation (e.g., information about points of interest, otherdescriptions, etc.) from the user regarding the viewing location and/orthe surrounding environment, such as for later use in presentation ofinformation regarding that viewing location and/or surroundingenvironment.

After block 415 is completed, the routine continues to block 420 todetermine if there are more viewing locations at which to acquireimages, such as based on corresponding information provided by the userof the mobile device, and/or based on automated tracking of viewinglocations visited (e.g., based on a predefined group of viewinglocations, based on acquiring at least one viewing location in each of apredefined group of rooms, etc.). If so, and when the user is ready tocontinue the process (if the device is carried by the user or itsmovement is otherwise controlled by the user), the routine continues toblock 422 to optionally initiate the capture of linking information(e.g., acceleration data and/or other data that changes as the devicemoves) during movement of the mobile device along a travel path awayfrom the current viewing location and towards a next viewing locationwithin the building interior. As described elsewhere herein, thecaptured linking information may include additional sensor data (e.g.,from one or more IMUs, or inertial measurement units, on the mobiledevice or otherwise carried by the user), as well as additional videoinformation, recorded during such movement. Initiating the capture ofsuch linking information may be performed in response to an explicitindication from a user of the mobile device or based on one or moreautomated analyses of information recorded from the mobile device. Inaddition, the routine may further optionally monitor the motion of themobile device in some embodiments during movement to the next viewinglocation, and provide one or more guidance cues to the user regardingthe motion of the mobile device, quality of the sensor data and/or videoinformation being captured, associated lighting/environmentalconditions, advisability of capturing a next viewing location,information about how to reach a defined next viewing location and/orabout when the defined next viewing location is reached, and any othersuitable aspects of capturing the linking information. Similarly, theroutine may optionally obtain annotation and/or other information (e.g.,information about points of interest, other descriptions, etc.) from theuser regarding the travel path, such as for later use in presentation ofinformation regarding that travel path or a resulting inter-panoramaconnection link. In block 424, the routine determines that the mobiledevice has arrived at the next viewing location (e.g., based on anindication from the user, based on the forward movement of the userstopping for at least a predefined amount of time, based on reaching apredefined spot for the next viewing location, etc.), for use as the newcurrent viewing location, and returns to block 415 in order to performthe viewing location image acquisition activities for the new currentviewing location.

If it is instead determined in block 420 that there are not any moreviewing locations at which to acquire image information for the currentbuilding or other structure, the routine proceeds to block 425 tooptionally analyze the viewing location information for the building orother structure, such as to identify possible additional coverage(and/or other information) to acquire within the building interior. Forexample, the ICA system may provide one or more notifications to theuser regarding the information acquired during capture of the multipleviewing locations and optionally corresponding linking information, suchas if it determines that one or more segments of the recordedinformation are of insufficient or undesirable quality, or do not appearto provide complete coverage of the building. After block 425, theroutine continues to block 435 to optionally preprocess the acquiredimages before their subsequent use for generating related mappinginformation, such as to perform an equirectangular projection for eachsuch image so that straight vertical data (e.g., the sides of a typicalrectangular door frame, a typical border between 2 adjacent walls, etc.)remains straight and in which straight horizontal data (e.g., the top ofa typical rectangular door frame, border between a wall and a floor,etc.) at a horizontal midline of the image remains straight but isincreasingly curved in the equirectangular projection image in a convexmanner relative to the horizontal midline as the distance increases inthe image from the horizontal midline. In at least some embodiments, anautomated analysis of some or all of the images may further be performedto determine corresponding information about the building interior, suchas types of rooms, points of interest in particular locations, etc. Inblock 477, the images and any associated generated or obtainedinformation is stored for later use, and optionally provided to an FMGMroutine for further analysis. FIGS. 5A-5B illustrate one example of aroutine for generating a floor map representation of a building interiorfrom the generated panorama information.

If it is instead determined in block 410 that the instructions or otherinformation recited in block 405 are not to acquire images and otherdata representing a building interior, the routine continues instead toblock 490 to perform any other indicated operations as appropriate, suchas any housekeeping tasks, to configure parameters to be used in variousoperations of the system (e.g., based at least in part on informationspecified by a user of the system, such as a user of a mobile device whocaptures one or more building interiors, an operator user of the ICAsystem, etc.), to obtain and store other information about users of thesystem, to respond to requests for generated and stored information,etc.

Following blocks 477 or 490, the routine proceeds to block 495 todetermine whether to continue, such as until an explicit indication toterminate is received, or instead only if an explicit indication tocontinue is received. If it is determined to continue, the routinereturns to block 405 to await additional instructions or information,and if not proceeds to step 499 and ends.

FIGS. 5A-5B illustrate an example embodiment of a flow diagram for aFloor Map Generation Manager (FMGM) System routine 500. The routine maybe performed by, for example, execution of the FMGM system 160 of FIG.1A, the FMGM system 379 of FIG. 3, and/or an FMGM system as discussedwith respect to FIGS. 1B-2L and elsewhere as described herein, such asto generate a computer model and optionally other mapping informationfor a defined area based at least in part on images of the area. In theexample of FIGS. 5A-5B, the generated mapping information includes a 3Dcomputer model and 2D floor map of a building, such as a house, but inother embodiments, other types of mapping information may be determinedand generated for other types of buildings and used in other manners, asdiscussed elsewhere herein.

The illustrated embodiment of the routine begins at block 505, whereinformation or instructions are received. The routine continues to block510 to determine whether the instructions received in block 505 indicateto generate mapping information for an indicated building, optionallyalong with associated information about the building, and if so theroutine continues to perform blocks 515-588 to do so, and otherwisecontinues to block 590.

In block 515, the routine determines whether image information isalready available for the building, or if such information instead needsto be acquired. If it is determined in block 515 that the informationneeds to be acquired, the routine continues to block 520 to acquire suchinformation, optionally waiting for one or more users or devices to movethroughout the building and acquire panoramas or other images atmultiple viewing locations in multiple rooms of the building, and tooptionally further analyze the images and/or metadata information abouttheir acquisition to interconnect the images, as discussed in greaterdetail elsewhere herein—FIG. 4 provides one example embodiment of an ICAsystem routine for performing such image acquisition. If it is insteaddetermined in block 515 that it is not necessary to acquire the images,the routine continues instead to block 530 to obtain existing panoramasor other images from multiple viewing locations in multiple rooms of thebuilding, optionally along with interconnection information for theimages and acquisition of metadata information related to movementbetween the viewing locations, such as may in some situations have beensupplied in block 505 along with the corresponding instructions.

After blocks 520 or 530, the routine continues to block 535 tooptionally obtain additional information about the building, such asfrom activities performed during acquisition and optionally analysis ofthe images, and/or from one or more external sources (e.g., onlinedatabases, information provided by one or more end users, etc.)—suchadditional information may include, for example, exterior dimensionsand/or shape of the building, additional images and/or annotationinformation acquired corresponding to particular locations within thebuilding (optionally for locations different from viewing locations ofthe acquired panorama or other images), etc.

After block 535, the routine continues to block 550 to determine, foreach room inside the building with one or more viewing locations andassociated acquired images, a room shape of the room for data in theimage(s) taken inside the room, and optionally a position within theroom of its viewing location(s), such as in an automated manner. Inblock 555, the routine further uses visual data in the images and/or theacquisition metadata for them to determine, for each room in thebuilding, any connecting passages in or out of the room, such as in anautomated manner. It will be appreciated that, while blocks 550-555, areillustrated as separate operations in this example, in some embodimentsa single analysis of the images may be performed to acquire or determinemultiple types of information, such as those discussed with respect toblocks 550-555.

In block 570, the routine then determines estimated positions of theroom shapes to create an initial 2D floor map, such as by connectinginter-room passages in their respective rooms, by optionally positioningroom shapes around determined viewing location positions (e.g., if theviewing location positions are inter-connected), and by optionallyapplying one or more constraints or optimizations. Such a floor map mayinclude, for example, relative position and shape information for thevarious rooms without providing any actual dimension information for theindividual rooms or building as a whole, and may further includemultiple linked or associated sub-maps (e.g., to reflect differentstories, levels, sections, etc.) of the building.

After block 570, the routine optionally performs one or more steps575-580 to determine and associate additional information with the floormap. In block 575, the routine optionally estimates the dimensions ofsome or all of the rooms, such as from analysis of images and/or theiracquisition metadata or from overall dimension information obtained forthe exterior of the building, and associates the estimated dimensionswith the floor map—it will be appreciated that if sufficiently detaileddimension information were available, a floor plan (e.g., architecturaldrawings, blue prints, etc.) may be generated from the floor map. Afterblock 575, the routine continues to block 580 to optionally associatefurther information with the floor map (e.g., with particular rooms orother locations within the building), such as additional images and/orannotation information. In block 585, the routine further estimatesheights of walls in some or all rooms, such as from analysis of imagesand optionally sizes of known objects in the images, as well as heightinformation about a camera when the images were acquired, and furtheruses such information to generate a 3D computer model of the building,with the 3D model and the floor map being associated with each other.

After block 585, the routine continues to block 588 to store thegenerated mapping information and optionally other generatedinformation, and to optionally further use the generated mappinginformation, such as to provide the generated 3D computer model and/or2D floor map for display on one or more client devices, provide thatgenerated information to one or more other devices for use in automatingnavigation of those devices and/or associated vehicles or otherentities, etc.

If it is instead determined in block 510 that the information orinstructions received in block 505 are not to generate mappinginformation for an indicated building, the routine continues instead toblock 590 to perform one or more other indicated operations asappropriate. Such other operations may include, for example, receivingand responding to requests for previously generated computer modelsand/or floor maps and/or other generated information (e.g., requests forsuch information for use by a BHP system in generating and presenting aGUI that simultaneously displays multiple types of information about abuilding in an integrated and coordinated manner, requests for suchinformation for display on one or more client devices, requests for suchinformation to provide it to one or more other devices for use inautomated navigation, etc.), obtaining and storing information aboutbuildings for use in later operations (e.g., information aboutdimensions, numbers or types of rooms, total square footage, adjacent ornearby other buildings, adjacent or nearby vegetation, exterior images,etc.), etc.

After blocks 588 or 590, the routine continues to block 595 to determinewhether to continue, such as until an explicit indication to terminateis received. If it is determined to continue, the routine returns toblock 505 to wait for and receive additional instructions orinformation, and otherwise continues to block 599 and ends.

FIG. 6 illustrates an example embodiment of a flow diagram for aBuilding Information Integrated Presentation (BIIP) system routine 600.The routine may be performed by, for example, execution of the BHPsystem 140 of FIG. 1A, the BHP system 340 of FIG. 3, and/or a BHP systemas discussed with respect to FIGS. 1B-2L and/or as described elsewhereherein, such as to generate and present a GUI that simultaneouslydisplays multiple types of information about a building in an integratedand coordinated manner. In the example of FIG. 6, the GUI uses multiplepanes to simultaneously present multiple types of information in anintegrated and coordinated manner, but in other embodiments, other typesof GUIs with other types of structures and layouts may be used topresent the same or similar types of information, as discussed elsewhereherein.

The illustrated embodiment of the routine begins in block 605, whereinformation or instructions are received. The routine continues to block610 to determine whether the instructions in block 605 are to presentintegrated information for an indicated building, such as in acorresponding GUI. If so, the routine continues to perform blocks615-650, and otherwise continues to block 690. In particular, if it isdetermined in block 610 that the instructions received in block 605 areto present integrated information for an indicated building, the routinecontinues to block 615 to obtain building information of multiple typesfor the indicated building, such as a 3D model of the building, imagesof the interior (and optionally, exterior) of the building, videos ofthe interior (and optionally, exterior) of the building, informationabout an interactive tour of a plurality of viewing/capture locationswithin the building interior (and optionally, exterior) at which imageand/or other information was captured, a 2-D floor plan, audio and/ortextual descriptions of particular locations or areas (e.g., rooms,points of interest, etc.), simulated and/or actual lighting information,information about surrounding buildings and/or vegetation and/or otherexterior aspects (vehicle traffic, foot traffic, noises, etc.), surfacesand/or areas available for virtual staging or otherwise for addingvirtual objects, etc.

After block 615, the routine continues to block 620 to determine aninitial GUI display for the building (e.g., to retrieve a preexistinginitial GUI to be used; to retrieve predefined information to add to theGUI; to generate the initial GUI, such as from one or more selectionsmade in the instructions received in block 605; etc.), such as a GUIwith three or more separate panes that simultaneously show at leastthree types of information about the building in a coordinated manner.In at least some embodiments, the at least three types of informationmay include one or more images and/or videos, at least some of the 3Dcomputer model, at least some of an interactive tour, etc., such as withall of the information focused on a common location or other area of thebuilding or other common feature or aspect. The GUI may further includevarious user-selectable controls (e.g., user-modifiable controls inwhich the user may modify a value or select from multiple options, otheruser-selectable controls in which functionality is toggled on and off orotherwise invoked upon selection, etc.) for display to an end user, suchas within one or more of the separate panes or instead in separate areasof the GUI (e.g., at the top, at the bottom, on the left and/or rightedges, etc.)—in at least some embodiments and situations, some or all ofthe user-selectable controls may be selected contextually based on otherinformation that is displayed in the GUI, such as to have a primary pane(e.g., that is larger than other secondary page) and to overlay thecontent shown in the primary pane with one or more user-selectablecontrols that are specific to that type of content. If the GUI is to bedisplayed on the same computer that is executing the routine 600, theroutine further generates a visual representation to supply to one ormore display devices or other output devices of that computer, andotherwise generates a visual representation for transmission to anotherclient device on which the visual representation of the GUI will bedisplayed. After block 620, the routine continues to block 625 toprovide the generated GUI visual representation to the computer deviceon which it will be displayed, and then waits for an event to trigger achange in the GUI (e.g., a user selection to modify the displayedinformation, a change in the underlying displayed information, an amountof time having elapsed since a last user input and/or a last displayactivity by the routine, etc.).

After block 625, the routine continues to block 630 to determine whetherthe event that has occurred with respect to block 625 indicates a changein the information to be displayed for the current building, and if socontinues to block 635 to determine an updated visual representation ofthe GUI to use as a new current GUI visual representation with thechanged information, and then returns to block 625 to initiate displayof the updated visual representation. The change may occur, for example,based on a user selection (e.g., of a user-selectable control in thedisplayed GUI) and/or a change in the underlying displayed information.In at least some embodiments and situations, the updated visualrepresentation again shows multiple types of information in multiple GUIpanes that all focus on a common area, location or other aspect of thebuilding (whether the same common area, location or other aspect, orinstead a different one) and that further include user-selectablecontrols corresponding to one or more panes of the GUI—in otherembodiments and situations, information displayed in the GUI at a giventime may include all or substantially all of the displayed area of theGUI, such as in a single pane and optionally being of a single type. Ifit is instead determined in block 630 that the event that occurred withrespect to block 625 did not cause a change in displayed information forthe current building, the routine continues to block 650 to determinewhether the event instead indicates to display information for a newbuilding, and if so the routine returns to block 615 to initiate adisplay of information for the new building in an updated version of theGUI.

If it is instead determined in block 610 that the instructions orinformation received in block 605 are not to present integratedinformation for an indicated building, the routine continues instead toblock 690 to perform one or more other indicated operations asappropriate. Such other operations may include, for example, receivingand responding to requests for previously generated GUI informationand/or for underlying information that may be used in a GUI, obtainingand storing information about buildings for use in later GUIpresentation operations (e.g., information from external sources, suchas one or more databases; information from end users, such as questionsabout an indicated building and/or an answer to an existing questionand/or images or other media captured in or around a building; etc.),performing automated analysis of available building information and/oractivities of one or more end users to determine patterns or otherwiseidentify additional information for subsequent use, etc.

After block 690, or if it is instead determined in block 650 that theevent is not to display information for a new building, the routinecontinues to block 695 to determine whether to continue, such as untilan explicit indication to terminate is received. If it is determined tocontinue, the routine returns to block 605 to wait for and receiveadditional instructions or information, and otherwise continues to block699 and ends.

Aspects of the present disclosure are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of thepresent disclosure. It will be appreciated that each block of theflowchart illustrations and/or block diagrams, and combinations ofblocks in the flowchart illustrations and/or block diagrams, can beimplemented by computer readable program instructions. It will befurther appreciated that in some implementations the functionalityprovided by the routines discussed above may be provided in alternativeways, such as being split among more routines or consolidated into fewerroutines. Similarly, in some implementations illustrated routines mayprovide more or less functionality than is described, such as when otherillustrated routines instead lack or include such functionalityrespectively, or when the amount of functionality that is provided isaltered. In addition, while various operations may be illustrated asbeing performed in a particular manner (e.g., in serial or in parallel,or synchronous or asynchronous) and/or in a particular order, in otherimplementations the operations may be performed in other orders and inother manners. Any data structures discussed above may also bestructured in different manners, such as by having a single datastructure split into multiple data structures and/or by having multipledata structures consolidated into a single data structure. Similarly, insome implementations illustrated data structures may store more or lessinformation than is described, such as when other illustrated datastructures instead lack or include such information respectively, orwhen the amount or types of information that is stored is altered.

From the foregoing it will be appreciated that, although specificembodiments have been described herein for purposes of illustration,various modifications may be made without deviating from the spirit andscope of the invention. Accordingly, the invention is not limited exceptas by corresponding claims and the elements recited by those claims. Inaddition, while certain aspects of the invention may be presented incertain claim forms at certain times, the inventors contemplate thevarious aspects of the invention in any available claim form. Forexample, while only some aspects of the invention may be recited asbeing embodied in a computer-readable medium at particular times, otheraspects may likewise be so embodied.

What is claimed is:
 1. A computer-implemented method comprising:selecting, by one or more computing systems, multiple types ofinformation about an indicated building to simultaneously show to a userin a displayed graphical user interface (GUI), wherein the multipletypes of information include at least one of an image or a video showingone or more locations associated with the indicated building, andinclude at least a portion of a floor map of the indicated building thatshows at least two-dimensional structural information for at least someof the indicated building and that indicates the one or more locationsfor the at least one of the image or the video; receiving, by the one ormore computing systems, one or more indications of one or moreadditional types of information to visually overlay on the at leastportion of the floor map that are based at least in part on dataacquired at the indicated building; and presenting, by the one or morecomputing systems, and via the displayed GUI, the multiple types ofinformation simultaneously in the displayed GUI, including to displaythe least one of the image or the video and to display the at leastportion of the floor map, and visually overlaying the one or moreadditional types of information on the displayed at least portion of thefloor map.
 2. The computer-implemented method of claim 1 wherein thepresenting of the multiple types of information in the displayed GUIincludes using multiple panes simultaneously in the displayed GUI suchthat the at least portion of the floor map is displayed in a first paneand the at least one of the image or the video is displayed in one ormore second panes.
 3. The computer-implemented method of claim 2 whereinat least one of the one or more second panes provides an interface foran interactive tour of a plurality of viewing locations in an interiorof the indicated building, and wherein the presenting of the multipletypes of information includes providing user-selectable controls thatmodify information shown in the displayed GUI, including to display, inthe at least one second pane, one or more user-selectable visualindicators of one or more viewing locations of the plurality.
 4. Thecomputer-implemented method of claim 1 wherein the presenting of themultiple types of information by the one or more computing systemsincludes transmitting at least the multiple types of information overone or more computer networks to a client computing device of the userfor display in the GUI on the client computing device, and wherein thereceiving of the one or more indications of the one or more additionaltypes of information is based at least in part by one or moreinteractions of the user with one or more user-selectable controlsdisplayed in the GUI.
 5. The computer-implemented method of claim 1wherein the one or more additional types of information includeinformation about at least one of public spaces for the indicatedbuilding or private spaces for the indicated building, and wherein thepresenting of the multiple types of information includes displayinginformation overlaid on the displayed at least portion of the floor mapthat includes visual indications of the at least one of the publicspaces or the private spaces.
 6. The computer-implemented method ofclaim 1 wherein the one or more additional types of information includeinformation about a type of room, and wherein the presenting of themultiple types of information includes displaying information overlaidon the displayed at least portion of the floor map that includes one ormore visual indications of at least one room in the indicated buildingthat is of the type of room.
 7. The computer-implemented method of claim6 wherein the one or more additional types of information furtherinclude multiple visual icons representing multiple room types, andwherein the presenting of the multiple types of information includesdisplaying information overlaid on the displayed at least portion of thefloor map that includes at least some of the multiple visual icons forrooms of respective types in the indicated building that are shown inthe displayed at least portion of the floor map.
 8. Thecomputer-implemented method of claim 1 wherein the one or moreadditional types of information include information about flow patternsof people within the indicated building, and wherein the presenting ofthe multiple types of information includes displaying informationoverlaid on the displayed at least portion of the floor map thatincludes visual indications of the flow patterns.
 9. Thecomputer-implemented method of claim 1 wherein the one or moreadditional types of information include personalized information that isspecific to the user about one or more spaces within the indicatedbuilding and that is based at least in part on one or more prior actionsof the user, and wherein the presenting of the multiple types ofinformation includes displaying information overlaid on the displayed atleast portion of the floor map that includes visual indications of thepersonalized information for the one or more spaces.
 10. Thecomputer-implemented method of claim 9 wherein the personalizedinformation specific to the user includes descriptive informationpreviously supplied by the user about at least one of the one or morespaces.
 11. The computer-implemented method of claim 1 furthercomprising: receiving, via one or more user interactions with thedisplayed GUI, instructions to display information about one or moredistances within the indicated building that are specified by the userwithin at least one of the displayed at least portion of the floor mapor the displayed at least one of the image or the video; and determiningvalues of the one or more distances within the indicated building, andwherein the one or more additional types of information include the oneor more distances, and wherein the presenting of the multiple types ofinformation includes displaying information overlaid on the displayed atleast portion of the floor map that includes visual indications of thedetermined values of the one or more distances.
 12. Thecomputer-implemented method of claim 11 wherein the one or more userinteractions about the one or more distances further indicate at leastone location within the indicated building to which the one or moredistances correspond, wherein the displaying of the information overlaidon the displayed at least portion of the floor map further includesplacing the visual indications of the determined values of the one ormore distances at one or more first positions on the displayed at leastportion of the floor map that correspond to the at least one location,and wherein the presenting of the multiple types of information furtherincludes displaying, simultaneously with the displaying of theinformation overlaid on the displayed at least portion of the floor map,second information that is overlaid on the displayed at least one of theimage or the video at one or more second positions corresponding to theat least one location and that includes visual indications of thedetermined values of the one or more distances.
 13. Thecomputer-implemented method of claim 1 wherein the one or moreadditional types of information include information about dimensions ofone or more rooms of the indicated building, and wherein the presentingof the multiple types of information includes displaying informationoverlaid on the displayed at least portion of the floor map thatincludes visual indications of the dimensions of the one or more rooms.14. The computer-implemented method of claim 13 wherein the displayingof the information overlaid on the displayed at least portion of thefloor map further includes placing the visual indications of thedimensions at one or more first positions on the displayed at leastportion of the floor map that correspond to the one or more rooms, andwherein the presenting of the multiple types of information furtherincludes displaying, simultaneously with the displaying of theinformation overlaid on the displayed at least portion of the floor map,second information that is overlaid on the displayed at least one of theimage or the video at one or more second positions corresponding to theone or more rooms and that includes visual indications of thedimensions.
 15. The computer-implemented method of claim 1 wherein theone or more additional types of information include information aboutchanges over time of an indicated type for the indicated building, andwherein the presenting of the multiple types of information includesdisplaying, over a period of time, information overlaid on the displayedat least portion of the floor map that includes visual indications ofthe changes over time of the indicated type.
 16. Thecomputer-implemented method of claim 15 wherein the changes over time ofthe indicated type include changes in traffic outside the indicatedbuilding over a second period of time that is longer than the period oftime, and wherein the displaying of the information over the period oftime includes using at least one of time-lapse display or acceleratedvideo display to show changes in traffic over the longer second periodof time during the period of time.
 17. The computer-implemented methodof claim 1 wherein the one or more additional types of informationinclude one or more images of at least one room of the indicatedbuilding that are projected on at least some walls of the at least oneroom, wherein the at least some of the indicated building shown in theat least two-dimensional structure includes the at least some walls andthe at least two-dimensional structure includes visual representationsof height of the at least some walls, and wherein the presenting of themultiple types of information includes displaying the one or more imagesprojected on at least some walls of the at least one room on thedisplayed at least portion of the floor map.
 18. Thecomputer-implemented method of claim 1 wherein the one or moreadditional types of information include information about audio recordedat the indicated building at one or more times, wherein the one or moreindications indicate at least one time of the one or more times, andwherein the presenting of the multiple types of information includespresenting information overlaid on the displayed at least portion of thefloor map that includes the audio recorded at the at least one time. 19.The computer-implemented method of claim 1 wherein the one or moreadditional types of information include information about lightingrecorded at the indicated building at one or more times, wherein the oneor more indications indicate at least one time of the one or more times,and wherein the presenting of the multiple types of information includesdisplaying information overlaid on the displayed at least portion of thefloor map that includes visual indications of the lighting recorded atthe indicated at least one time.
 20. The computer-implemented method ofclaim 1 wherein the indicated at least one time is at nighttime and thelighting recorded at the at least one time is from one or more interiorlights in the indicated building, and/or the indicated at least one timeis a time-of-year.
 21. A system comprising: one or more hardwareprocessors of one or more computing systems; and one or more memorieswith stored instructions that, when executed by at least one of the oneor more hardware processors, cause the one or more computing systems toperform automated operations including at least: selecting multipletypes of information about a location associated with an indicatedbuilding to show together to a user in a displayed graphical userinterface (GUI), wherein the multiple types of information include atleast a portion of a floor map of the indicated building that showsstructural information for at least some of the indicated building andthat indicates the location, and include a visual display of datacaptured at the location; receiving one or more indications of one ormore additional types of information about the indicated building tovisually overlay on information displayed in the GUI; and providing themultiple types of information and the one or more additional types ofinformation for display together in the displayed GUI, including todisplay the at least portion of the floor map and to provide the visualdisplay of the captured data, and to visually overlay the one or moreadditional types of information in the displayed GUI on at least one ofthe displayed at least portion of the floor map or the visual display ofthe captured data.
 22. The system of claim 21 wherein the providing ofthe multiple types of information and the one or more additional typesof information includes transmitting the multiple types of informationand the one or more additional types of information over one or morecomputer networks to a client computing device of the user for displayin the GUI on the client computing device, and wherein the receiving ofthe one or more indications of the one or more additional types ofinformation is based at least in part by one or more interactions of theuser with one or more user-selectable controls displayed in the GUI. 23.The system of claim 21 wherein the providing of the multiple types ofinformation and the one or more additional types of information includescausing display of provided information using multiple panes in thedisplayed GUI, including to display the at least portion of the floormap in a first pane and to simultaneously display the visual display ofthe captured data in one or more second panes.
 24. The system of claim23 wherein at least one of the one or more second panes provides aninterface for an interactive tour of a plurality of viewing locations inan interior of the indicated building, and wherein the providing of themultiple types of information and the one or more additional types ofinformation includes providing user-selectable controls in the displayedGUI that modify information shown in the displayed GUI, including tocause display in the at least one second pane of one or moreuser-selectable visual indicators of one or more viewing locations ofthe plurality.
 25. The system of claim 21 wherein the one or moreadditional types of information include personalized information that isspecific to the user about one or more spaces within the indicatedbuilding and that is based at least in part on one or more prior actionsof the user, and wherein the providing of the multiple types ofinformation and the one or more additional types of information includescausing visual indications of the personalized information for the oneor more spaces to be displayed in the GUI on at least one of thedisplayed at least portion of the floor map or the visual display of thecaptured data.
 26. The system of claim 21 wherein the one or moreadditional types of information include descriptive informationpreviously supplied by the user about the indicated building, andwherein the providing of the multiple types of information and the oneor more additional types of information includes causing visualindications of the descriptive information to be displayed in thedisplayed GUI.
 27. The system of claim 21 wherein the locationassociated with the indicated building includes one of multiple rooms ofthe indicated building, wherein the visual display of data captured atthe location includes at least one of an image or a video that shows atleast some of an interior of the one room, and wherein the displayed atleast portion of the floor map further shows three-dimensionalstructural information for at least some of the one room.
 28. The systemof claim 21 wherein the location associated with the indicated buildingis at least some of a first room of the building, and wherein theautomated operations further include: identifying multiple pieces ofdata captured in the first room, including at least one first image thatis shown in the GUI as part of the visual display of data captured atthe location; receiving, via one or more user interactions with thedisplayed GUI, instructions to change the at least one first image to atleast one other of the multiple pieces of data captured in the firstroom; and presenting, to the user via the displayed GUI, updatedinformation that includes the at least one other piece of data capturedin the first room, and that includes one or more indications on thedisplayed at least portion of the floor map of one or more locationsassociated with the at least one other piece of data captured in thefirst room.
 29. The system of claim 21 wherein the floor map furtherincludes three-dimensional structural information for the at least someof the indicated building that includes walls and windows and doorways,and wherein the automated operations further include providing the atleast portion of the floor map to cause a display of the at leastportion of the floor map in the displayed GUI that shows thethree-dimensional structural information and that does not includevisual indications of contents within an interior of the at least someof the indicated building that are not part of its structure.
 30. Thesystem of claim 29 wherein the providing of the at least portion of thefloor map and the display of the at least portion of the floor map inthe displayed GUI that shows the three-dimensional structuralinformation are performed before the receiving of the one or moreindications of the one or more additional types of information, andwherein the providing of the multiple types of information and the oneor more additional types of information for display together in thedisplayed GUI causes an update of the display at least portion of thefloor map to include a visual overlay of the one or more additionaltypes of information.
 31. The system of claim 21 wherein the locationassociated with the indicated building includes at least one areaoutside of the indicated building, wherein the at least portion of thefloor map of the indicated building includes visual representations ofthe at least one area, wherein the one or more additional types ofinformation include at least one of a yard adjacent to the indicatedbuilding or vegetation around the indicated building or one or moresupplemental structures for the indicated building or one or more otherbuildings visible from the indicated building, and wherein visuallyoverlaying the one or more additional types of information in thedisplayed GUI includes using at least one of one or more representativegeometrical shapes or one or more images of the at least one area. 32.The system of claim 21 wherein the location associated with theindicated building is within an interior of the indicated building, andwherein the one or more additional types of information includesimulated lighting for the location for at least one of an indicatedtime-of-day, or an indicated time-of-year, or daylight entering theinterior, or one or more interior lights at night.
 33. The system ofclaim 21 wherein the one or more additional types of information includeat least one of one or more images supplied by one or more end users whovisit the indicated building, or one or more descriptions supplied byone or more end users who view the indicated building, or one or moreanswers supplied by one or more end users to one or more questions aboutthe indicated building supplied by one or more other end users, orinformation about weather patterns typical for an area of the indicatedbuilding, or information about a geographical location of the indicatedbuilding, or information about a geographical orientation of theindicated building, or an audio description of an area associated withthe location.
 34. A non-transitory computer-readable medium havingstored contents that cause one or more computing systems to performautomated operations including at least: selecting, by the one or morecomputing systems, multiple types of information about a locationassociated with an indicated building to show together to a user in adisplayed graphical user interface (GUI), wherein the location includesone of multiple rooms of the indicated building, and wherein themultiple types of information include at least a portion of a floor mapof the indicated building that shows three-dimensional structuralinformation for the one room including walls and windows and doorways,and further include a visual display based on data captured at thelocation; and providing, by the one or more computing systems, themultiple types of information for display together in the displayed GUI,including to cause a display of the at least portion of the floor mapthat shows the three-dimensional structural information and that doesnot include visual indications of contents within an interior of the oneroom that are not part of its structure, and to provide the visualdisplay of the captured data.
 35. The non-transitory computer-readablemedium of claim 34 wherein the visual display based on the data capturedat the location includes at least one of a first image or a first videothat shows at least some of an interior of the one room, and wherein thestored contents include software instructions that, when executed, causethe one or more computing systems to perform further automatedoperations including at least: identifying multiple pieces of datacaptured in the one room, including the at least one of the first imageor the first video; receiving, via one or more user interactions withthe displayed GUI, instructions to change the visual display in thedisplayed GUI to at least one other of the multiple pieces of datacaptured in the one room; and presenting, to the user via the displayedGUI, updated information that includes the at least one other piece ofdata captured in the one room, and that includes one or more indicationson the displayed at least portion of the floor map of one or morepositions within the one room associated with the at least one otherpiece of data captured in the one room.
 36. The non-transitorycomputer-readable medium of claim 34 wherein the stored contents includesoftware instructions that, when executed, cause the one or morecomputing systems to perform further automated operations including atleast, after the providing of the multiple types of information fordisplay together, receiving one or more indications of one or moreadditional types of information about the indicated building to visuallyoverlay on information displayed in the GUI, and presenting, to the uservia the displayed GUI, updated information that includes the one or moreadditional types of information overlaid on the displayed at leastportion of the floor map.
 37. The non-transitory computer-readablemedium of claim 36 wherein the one or more additional types ofinformation include personalized information that is specific to theuser about the one room and that is based at least in part on one ormore prior actions of the user, and wherein the presenting of theupdated information includes causing visual indications of thepersonalized information to be displayed in the GUI.